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SUMMARY 
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Simulations  of  ground  mapping  radar  are  generated  from  Defense  Mapping 
Agency  Digital  Feature  Analysis  Data  (DFAD)  products  which  were  developed  to 
support  real  beam  ground  mapping  radar.  Criteria  for  inclusion  of  an  object 
within  a  given  level  of  DFAD  are  radar  significance,  height,  and  size.  In 
general,  Level  1  includes  objects  100m  and  larger  and  Level  2  includes  objects 
30m  and  larger.  A  prototype  DFAD  incorporating  features  as  small  as  10m  (Level 
X)  was  developed  to  support  the  higher  resolution  Synthetic  Aperture  Radar 
(SAR);  however,  the  minimum  data  base  requirements  for  simulating  SAR  have  not 
been  determined.  The  present  report  describes  three  studies  conducted  to 
determine  these  requi rements .  In  the  first  study,  B-1B  and  F-15E  radar 
operators  rated  SAR  simulations  produced  from  Level  2  DFAD  (30m)  and  Level  X 
(10m),  plus  two  experimental  data  bases  with  15m  and  20m  capture  criteria. 
Eighty-two  percent  of  SAR-experienced  subjects  found  the  15m  data  acceptable. 
Simulations  produced  from  Level  2  depicted  areas  of  high  reflector  density  as  a 
single  feature  with  uniform  brightness  and  were  acceptable  to  only  18%  of  the 
subjects.  A  study  of  the  Radar  Scope  Interpretation  (RSI)  cues  used  by  B-1B 
Offensive  Systems  Officers  (OSOs)  found  that  roads  and  rivers  were  critical 
cues,  whereas  individual  structures  were  used  only  in  the  immediate  vicinity  of 
the  aimpoint.  .Based  on  these  findings,  it  was  predicted  that  areas  of  high 
feature  densityxcould  be  depicted  generically  without  affecting  task 
performance.  A  rbird  study  compared  R-1B  0S0  performance  on  a  navigation 
update  task  using  TOm  and  15m  feature  data  and  generically  enhanced  30m  (Level 
2)  and  100m  (Level  1)  data.  This  study  found  that  task  performance,  operator 
confidence,  and  rated  acceptability  using  10m,  15m,  or  enhanced  30m  data  were 
not  significantly  different.  The  authors  concluded  that  the  density  of  task- 
critical  features  in  Level  2  DFAD  is  sufficient  to  simulate  SAR  but  that  high 
density  areas  must  be  enhanced  to  create  acceptable  simulation  fidelity. 
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I.  INTRODUCTION 


Digital  Radar  Landmass  Simulators  (DRUMS)  generate  simulated  radar  ground 
maps  based  on  a  digital  model  of  the  earth's  surface.  This  model  is  often 
created  using  the  Defense  Mapping  Agency  (DMA)  Digital  Landmass  System  (DLMS) 
data.  DLMS  data  consist  of  two  files  for  a  given  area:  Digital  Terrain 
Elevation  Data  (DTED)  and  Digital  Feature  Analysis  Data  (DFAD).  Measurements  of 
terrain  height  above  sea  level  are  spaced  at  regular  intervals,  typically  300nm, 
and  describe  surface  contours  only.  DFAD  consist  of  descriptions  of  objects  on 
the  surface  such  as  vegetation,  cultivated  fields,  bodies  of  water,  roads, 
cities,  and  structures.  DMA  produces  these  data  files  at  differing  levels  of 
detail  for  different  applications.  Level  1  DFAD  cover  large  expanses  of  the 
earth's  surface,  with  relatively  large  minimum  size  requirements  for  portrayal 
of  planimetric  features  (DMA  1983,  p.  1).  Level  2  DFAD  cover  small  areas  of 
interest  and  have  smaller  minimum  size  requirements  for  portrayal  of 
planimetric  features  (DMA  1983,  p.  1). 

These  products  were  developed  in  the  1970s  to  support  simulations  of 
existing  real  beam  ground  mapping  radars  such  as  the  B-52,  KC-135,  and  C - 1 30 
aircraft.  These  radars  produce  ground  maps  of  relatively  large  areas  and 
cannot  re solve  small  features  individually.  Newer  radars  such  as  the  Synthetic 
Aperture  Radars  (SARs)  in  the  B-1B  and  F-15E,  however,  produce  ground  maps  of 
areas  as  small  as  4000'  x  4000'  with  a  resolution  of  10'  or  less.  Current  DFAD 
products  do  not  contain  sufficient  object  density  to  support  realistic 
simulations  of  these  SARs. 

Reducing  the  minimum  size  required  for  inclusion  within  the  DFAD  file  would 
increase  the  number  of  features  within  the  data  base  but  would  also  increase 
DFAD  production  costs.  This  report  describes  three  studies  evaluating  the  data 
density  requirement  for  simulating  SAR.  The  objectives  were:  (a)  to  determine 
the  minimum  density  of  ground  truth*  data  required  to  adequately  simulate  SAR, 
(b)  to  identify  the  critical  ground  features  used  in  SAR  image  interpretation, 
and  (c)  to  investigate  the  utility^of  selectively  replacing  ground  truth 
information  with  synthetic  imagery^  in  order  to  reduce  cost  without  affecting 
training  utility. 


Ground  truth  denotes  that  each  object  in  the  data  base  accurately  reflects 
an  object  on  the  ground  in  terms  of  position,  size,  height,  orientation, 
feature  type,  and  surface  material. 

2 

Synthetic  imagery  denotes  that  objects  in  the  data  base  may  not 
necessarily  represent  specific  objects  on  the  ground;  features  such  as  houses, 
warehouses,  or  secondary  roads  may  be  inserted  into  the  data  base  in  areas 
where  these  features  are  located  without  regard  for  matching  specific  objects 
on  the  ground. 
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Background 

In  the  late  1970s  the  Air  Force  and  DMA  realized  that  current  DLMS  products 
could  not  support  SAR  simulations.  In  response,  DMA  created  a  prototype  high- 
resolution  DFAD  called  Level  X.  The  capture  criterion  (i.e.,  the  smallest 
horizontal  dimension  for  individual  feature  portrayal)  for  Level  X  was  10m. 

This  criterion  was  selected  on  the  basis  of  an  engineering  assessment  of  SAR 
capabilities  and  lessons  learned  from  simulating  real  beam  radar.  In  general, 
features  smaller  or  closer  together  than  the  nominal  capture  criterion  will  not 
be  individually  portrayed  in  the  DFAD  file.  Radar  significant  features  which 
are  smaller  than  the  criterion,  however,  are  individually  portrayed  on  an 
exception  basis. 

Figures  1  through  3  are  DFAD  manuscripts  of  Greybull,  Wyoming  at  Levels  1, 
2,  and  X.  The  capture  criterion  is  100m  for  Level  1,  30m  for  Level  2,  and 
10m  for  Level  X.  Level  1  contains  59  features  while  Level  2  contains  281  and 
Level  X  contains  1,389.  Figure  4  contains  an  aerial  photograph  of  Greybull  for 
comparison. 
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Figure  1.  Plot  of  Digital  Feature  Analysis  Data  for  Greybull,  Wyoming 
~  ~~  at  Level  1 . 
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Figure  3. 


Plot  of  i  .ital  Feature  Analysis  Data  for  Greybull ,  Wyoming, 
at  Lev;'! 
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4_.  Fixpoint  Graphic  Card  for  Greybull,  Wyoming,  Including 
1.25  x  1.25  nm  aerial  photograph. 


An  engineering  analysis  of  SAR  simulations  produced  from  Level  X  was 
conducted  by  The  Analytical  Sciences  Corporation  (TASC)  in  1985.  This  analysis 
concluded  that  Level  X,  with  minor  modifications  such  as  the  addition  of  roof 
clutter  on  large  buildings  and  the  elimination  of  individual  trees,  would 
support  acceptable  SAR  simulations.  The  TASC  analysis  did  not  address  the 
issue  of  whether  a  data  base  less  dense  than  Level  X  would  adequately  simulate 
SAR,  nor  were  subject-matter  experts  (SMEs)  from  the  using  commands  consulted. 
In  1985,  the  Operations  Training  Division  of  the  Air  Force  Human  Resources 
Laboratory  (AFHRL/OT)  and  the  Aeronautical  Systems  Division  (ASD/ENETV)  began 
work  to  determine  the  minimum  data  base  density  required  for  simulating 
current-generation  SAR. 


Overview 


The  three  studies  described  in  the  present  report  were  designed  to  do  what 
the  earlier  analysis  did  not;  namely,  to  consult  SAR  operators  and  instructors 
in  order  to  determine  the  minimum  data  set  required  to  adequately  simulate  SAR. 
The  three  studies  consisted  of:  (a)  subject-matter  expert  evaluations  of 
simulated  SAR  generated  from  different  data  bases,  (b)  analysis  of  Radar  Scope 
Interpretation  (RSI)  cues  used  by  SAR  operators,  and  (c)  SAR  operator  task 
performance  using  SAR  simulations  generated  from  different  data  bases. 

The  first  study  examined  density  requirements  by  having  both  SAR- 
experienced  and  non-SAR-experienced  radar  operators  evaluate  SAR  simulations 
produced  at  four  levels  of  data  base  density.  The  evaluation  consisted  of 
having  the  radar  operators  examine  photographs  of  the  simulations  and  rate  the 
acceptability  of  each  level  in  terms  of  providing  the  minimum  amount  of  detail 
necessary  for  adequate  simulation.  The  images  were  created  directly  from  the 
different  data  bases  without  enhancements  or  modifications.  This  study 
suggested  that  something  less  than  Level  X  might  be  adequate  for  SAR 
simulation. 

The  second  study  was  an  analysis  of  the  types  of  ground  features  used  by 
SAR  operators  as  RSI  cues.  Aerial  photographs  of  115  scenes  were  inspected  by 
B-1B  Offensive  Systems  Officers  (OSOs).  The  OSOs  indicated  which  ground 
features  they  would  use  as  RSI  cues,  or  pointers,  in  order  to  locate  a 
specified  aimpoint.  Analysis  of  these  data  indicated  that  roads,  rivers,  and 
railroad  tracks  were  critical  and  that  many  small  features  might  be  depicted 
generically  without  affecting  operator  performance. 

The  third  study  was  an  assessment  of  B-1B  0S0  task  performance  using 
simulated  SAR  images  produced  from  differing  levels  of  data  base  density. 

These  included  Level  X  which  contains  a  very  high  density  of  ground  truth 
information  and  a  version  of  Level  2  which  was  enhanced  using  synthetic  imagery 
to  generically  depict  areas  of  high  feature  density.  Since  the  RSI  cue 
analysis  indicated  that  individual  features  within  dense  areas  were  not  used  as 
pointers,  it  was  predicted  that  these  areas  did  not  require  detailed  ground 
truth  to  support  task  performance.  Results  of  this  study  showed  that  0S0  task 
performance  using  the  enhanced  Level  2  was  not  significantly  different  from 
that  for  Level  X. 

These  three  studies  demonstrated  that  (a)  the  aimpoint  plus  task-critical 
RSI  cues  such  as  roads  and  rivers  must  be  accurately  portrayed  in  order  for  SAR 
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operators  to  perform  the  required  tasks,  and  (b)  the  entire  simulated  scene 
must  be  similar  to  an  actual  SAR  scene  or  it  will  not  be  accepted  by  the 
operators.  The  studies  also  showed  that  some  small  features  can  be  depicted 
generically  without  reducing  training  effectiveness. 

II.  DENSITY  REQUIREMENTS  STUDY 
Research  Methods 


Materials 


Simulated  SAR  images  were  created  from  DFAD  files  containing  different 
individual  feature  densities.  These  data  bases  were  Level  2  and  Level  X  DFAD 
with  capture  criteria  of  30m  and  10m,  respectively,  and  two  experimental  data 
bases  called  Level  Y  and  Level  Z  with  capture  criteria  of  15m  and  20m, 
respectively.  To  be  individually  depicted  in  the  data  base,  features  must  have 
at  least  one  horizontal  dimension  greater  than  the  capture  criterion. 
Collections  of  structures  such  as  houses  which  are  smaller  and  more  closely 
spaced  than  the  capture  criterion  for  a  given  data  base  level  were  included  as 
a  single  areal  feature^  which  incorporated  the  outline  of  the  area. 

Level  X  DFAD  were  provided  by  DMA  for  four  locations:  Scott  AFB  and  three 
fixpoints  in  the  Strategic  Training  Range  Complex  (STRC).  The  Level  Y  and 
Level  Z  data  bases  were  created  by  manually  filtering  the  Level  X  data  to  15m 
and  20m  capture  criteria.  Level  2  DFAD  were  also  provided  by  DMA  for  Scott 
AFB.  Since  Level  2  data  were  not  available  for  the  STRC  locations,  these  data 
were  produced  by  manually  filtering  the  Level  X  data  to  conform  to  the  Level  2 
(2nd  edition)  DLMS  specification. 

SAR  simulations  were  produced  from  these  16  data  bases  (4  densities  at  4 
locations)  using  a  Singer-Link  research  and  development  SAR  simulator. 
Simulations  were  produced  for  each  location  and  data  base  density  with  a  1.2nm 
x  1.2nm  patch  (201  resolution)  and  a  .67nm  x  .67nm  patch  (10'  resolution). 

These  patch  sizes  and  resolutions  are  comparable  to  B-1B  and  F-15E  SAR 
capabilities.  The  radar  effects  included  in  the  simulations  were  shadow, 
noise,  leading  edge  brightening,  and  star  effects.  Texture  patterns  were 
overlaid  on  only  ground,  grass,  and  tree  areas.  Terrain  relief,  far  shore 
brightening,  and  wind  effects  were  not  simulated.  Areal  features  containing 
numerous  small  reflectors  were  not  enhanced.  Images  for  the  three  STRC  points 
were  produced  from  a  simulated  range  of  8nm  and  altitude  850'  above  ground 
level  (AGL),  producing  a  1-degree  grazing  angle.  For  Scott  AFB,  altitude  was 
1,700'  AGL,  producing  a  2-degree  grazing  angle. 


An  areal  feature  is  any  topographic  feature  such  as  sand,  swamp,  residential, 
or  industrial  which  extends  over  an  area.  In  digital  mapping,  an  areal  feature 
is  enclosed  by  a  delimiting  line,  and  is  assigned  a  single  unique  character 
(e.g. ,  residential ) . 
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Each  image  was  photographed  and  printed  in  uiack-and-white;  the  radar 
image  was  approximately  5.5"  x  5.5"  on  a  black  background.  Image  resolution 
was  400  x  400  pixels.  Features  within  the  four  scenes  were:  (a)  Scott  AFB-- 
runways,  hangars,  shops,  and  residential  housing;  (b)  STRC  Point  29--a  low 
cultural  detail  area  including  a  river,  a  freeway,  and  some  fence  lines;  (c) 
STRC  point  4--a  small  town  with  a  freeway  on  an  embankment,  and  a  power  line; 
and  (d)  STRC  Point  13--an  isolated  farm  consisting  of  several  buildings  and 
some  fences.  These  four  sets  of  images  were  labeled,  respectively:  (a)  Scott 
AFB,  (b)  River,  (c)  Town,  and  (d)  Farm.  Figures  5  through  8  show  the  20' 
resolution  SAR  simulations  for  Scott  AFB  at  each  data  base  level. 


Figure  5.  Simulated  SAR  Image  of  Scott  AFB  Used  in  the  Density  Require¬ 
ments  Study  Generated  from  level  2  (30nm)  Data.  Only  the  soil 
and  tree  areas  have  been  textured;  no  enhancements  have  been 
added  to  other  features.  Patch  size  is  1.25  x  1.25nm; 
simulated  radar  resolution  is  20'. 
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Figure  8.  Simulated  SAR  Image  of  Scott  AFB  Used  in  the  Density 
Requirements  Study  Generated  from  Level  X  (10m)  Data. 


Subjects 


Data  were  collected  from  two  groups  of  radar  operators.  The  non-SAR- 
experienced  consisted  of  30  B-52  Instructor  Radar-Navigators  whose  experience 
was  limited  to  real  beam  ground  mapping  radars.  The  SAR-experienced  group  of 
subjects  included  14  B-1B  OSOs  and  3  F-15E  Weapons  Systems  Officers. 

Procedure 


Subjects  participated  individually  or  in  groups  on  an  as-available  basis 
within  their  squadrons.  The  purpose  of  the  study  was  explained  and  the 
characteristics  of  the  test  images  described.  For  each  of  the  four  fixpoints, 
subjects  examined  the  10'  and  20'  resolution  images  and  rated  the  acceptability 
of  each  data  base  level.  It  was  pointed  out  that  where  absolute  ground  truth 
was  not  required,  synthetic  imagery  corresponding  to  the  general  character¬ 
istics  of  an  area  (such  as  residential  housing)  could  be  added  to  the  images. 
This  type  of  enhancement  would  be  generically  correct  but  not  specific  to  a 
given  area.  The  images  were  spread  out  on  a  table  top;  subjects  examined  and 
compared  them  without  time  constraints  and  rated  on  questionnaires  whether  each 
image  contained  the  "minimum  acceptable  ground  truth  information  necessary  for 
simulating  SAR  for  any  purpose,  including  training,  mission  planning,  and 
mission  rehearsal."  The  exact  aimpoint  within  each  scene  was  not  specified  nor 
were  aerial  photographs  or  charts  provided.  Subjects  were  encouraged  to  offer 
comments  or  suggestions  about  the  images  or  about  the  test  procedures. 
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Results  and  Discussion 


Table  1  and  Table  2  show  the  number  of  subjects  rating  a  given  level  of 
detail  as  "minimally  acceptable"  for  the  non-SAR-experienced  and  SAR- 
experienced  groups,  respectively.  A  log-linear  analysis  comparing  the  marginal 
totals  for  the  SAR-experienced  and  the  non-SAR*experi enced  groups  shows  that 
the  two  groups  rated  the  images  differently  (X^(17)=30.0,  p<.05).  The  two 
groups  will  therefore  be  discussed  separately. 


Table  1.  Minimum  Acceptable  Data  Base  Level  Rated 
30  Non-SAR-Experi enced  Radar  Operators 

Level 


X  (10m)  Y  (15m)  Z  (20m)  2  (30m) 


Scott  AFB 

River 

Town 

Farm 

Total 


3  4  15 

2  26  2 

1  4  19 

1  9  14 

7  (6%)  43  (36%)  50  (42%) 
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0 

6 

6 

20  (16%) 


Examination  of  the  data  from  the  non-SAR-experienced  group,  Table  1,  shows 
that  for  the  River  fixpoint  2  (6.6%)  of  the  subjects  rated  Level  X  as  minimally 
acceptable,  2  (6.6%)  rated  Level  Z  as  minimally  acceptable,  and  26  of  the  30 
(86.6%)  rated  Level  Y  as  minimally  acceptable.  The  pattern  of  responses  for 
the  other  three  fixpoints  is  markedly  different  from  that  of  the  River 
fixpoint. 


Table  2.  Minimum  Acceptable  Data  Base  Level  Rated  by  17 
SAR-Experienced  Radar  Operators 

Level 


X  (10m)  Y  (15m)  Z  (20m)  2  (30m) 


Scott  AFB 
Ri  ver 
Town 
Farm 
Total 


4 
2 
0 

5 

11  (16%) 


5 

14 

7 

7 

33  (49%) 


7 

1 

7 

4 

19  (28%) 


1 

0 

3 

1 

5  (7%) 


Inspection  of  the  data  from  the  17  SAR-experienced  subjects,  Table  2, 
reveals  that  for  the  River  fixpoint,  2  (12%)  rated  Level  X  as  minimally 
acceptable,  14  (82%)  chose  Level  Y,  and  1  (6%)  chose  Level  Z.  The  pattern  of 
responses  for  the  other  three  fixpoints  is  different  than  that  for  River. 
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The  River  fixpoint  image  includes  very  little  cultural  detail;  there  are  a 
river,  a  freeway,  and  some  fence  lines.  When  the  data  base  was  filtered 
from  Level  X  to  Level  Z,  the  river  itself  was  changed  from  an  areal  feature  to 
a  narrow  line  feature  which  does  not  appear  on  the  photographs  for  Level  Z 
or  Level  2.  The  river  therefore  disappears  between  Level  Y  and  Level  Z, 
leaving  very  few  features  in  these  images.  Subjects'  comments  were 
consistently  critical  of  the  absence  of  the  river  in  the  less  detailed  images 
since  rivers  are  often  used  as  RSI  cues  or  aimpoints.  As  a  result,  subjects 
required  that  the  river  be  present  in  the  images  and,  since  little  detail  was 
added  in  Level  X  over  Level  Y,  85%  of  all  subjects  rated  Level  Y  as  containing 
the  minimally  acceptable  level  of  detail.  Since  the  results  for  the  River 
fixpoint  were  driven  by  the  presence  or  absence  of  a  single  feature  and  not  the 
overall  level  of  detail,  the  data  for  this  set  of  images  were  considered 
aberrant  and  omitted  from  further  analyses.  Marginal  totals  for  Scott  AFB, 
Town,  and  Farm  are  presented  in  Table  3. 

Table  3.  Minimum  Acceptable  Data  Base  Level  for  SAR-Experienced 
and  Non-SAR-Experienced  Subjects  for  Scott  AFB,  Town, 
and  Farm 

Level 


non-SAR- 

experienced  5  (6%)  17  (19%)  48  (53%)  20  (22%) 

SAR- 

experienced  9  (18%) _  19  (37%) _ 18 J3_ 5% ) _ 5  (10%) 

Since  the  feature  density  decreased  from  Level  X  to  Level  2,  a  subject 
selecting  a  given  data  base  level  as  minimally  acceptable  would  rate  more  dense 
levels  as  also  acceptable.  Among  the  non-SAR-experienced  subjects,  22%  found 
Level  2  acceptable.  When  the  density  was  increased  to  Level  Z,  an  additional 
53%  or  a  total  of  75%  of  the  subjects  rated  Level  Z  as  acceptable.  However, 
among  the  SAR-experienced  subjects,  only  10%  found  Level  2  acceptable  and 
increasing  density  to  Level  Z  added  only  35%,  for  a  total  of  45%.  That  is, less 
than  half  of  the  SAR-experienced  operators  were  satisfied  with  Level  Z. 
Increasing  the  density  to  Level  Y  adds  37%  ,  for  a  total  of  82%.  It  can  be  seen, 
therefore,  that  demand  for  detail  in  radar  simulations  increases  with 
experience  in  using  high-resolution  radar.  It  can  also  be  seen  that  a  data 
base  density  greater  than  Level  Z  (20m)  is  required  in  order  to  satisfy  at  least 
half  of  the  SAR-experienced  operators. 

In  addition  to  the  ratings,  subjects  were  asked  to  offer  comments  about  the 
simulations  and  the  task.  The  comments  indicated  that  the  task  of  selecting 
the  "minimum  acceptable  density"  was  somewhat  ambiguous  because  the  aimpoint 
was  not  specified.  For  example,  if  a  B-l  0S0  were  asked  to  center-aim  a 
cluster  of  returns,  then  returns  from  small  features  would  not  be  required. 
Alternatively,  returns  from  small  features  would  be  required  if  the  specified 
aimpoint  were  an  isolated  reflector  or  an  individual  reflector  within  a  cluster 
of  objects.  The  variable  responses  for  Scott  AFB,  Town,  and  Farm  reflect 
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differences  of  opinion  as  to  what  the  appropriate  aimpoints  should  be. 

Subjects  who  rated  Level  X  as  minimally  acceptable  for  Scott  AFB  or  Farm 
commented  that  they  would  choose  very  small  aimpoints  such  as  the  corner  of  a 
building  or  an  intersection  of  two  fences;  subjects  who  would  choose  larger 
targets  such  as  the  intersection  of  runways  or  who  would  center-aim  the  cluster 
of  buildings  in  Farm  specified  less  detail  as  minimally  acceptable.  Overall, 
the  rated  level  of  acceptability  depended  on  both  the  size  and  density  of 
features.  High  density  data  bases  were  required  when  small  features  would  be 
individually  detected  in  a  SAR  image;  however,  they  were  not  required  when  the 
features  were  so  close  together  that  no  single  object  would  be  picked  as  an 
aimpoint.  Larger  landmarks  within  a  densely  packed  area,  such  as  the  lines  of 
communication  in  the  housing  area  at  Scott  AFB  (see  Figures  5  and  6),  were 
demanded  regardless  of  feature  size.  Finally,  several  SAR-experienced  subjects 
commented  that  as  they  gained  experience  in  using  SAR  in  their  aircraft  they 
were  looking  for  smaller  and  smaller  aimpoints  which  were  typically  isolated 
reflectors  such  as  fence  intersections  and  microwave  towers. 

Several  suggestions  were  offered  regarding  future  studies.  In  addition  to 
specification  of  aimpoints,  subjects  wanted  more  information  about  what 
features  were  really  on  the  ground  and  what  they  would  look  like  on  SAR.  For 
each  fixpoint,  subjects  would  like  to  see  an  actual  SAR  image  or  a  vertical 
photograph  of  the  ground  patch  being  simulated. 

The  following  conclusions  are  supported  by  the  data  collected  in  this 
study: 


1.  With  experience,  operators  tend  to  select  smaller  objects  and  features 
as  aimpoints.  These  features  tend  to  be  isolated  reflectors  or  small  parts  of 
larger  structures.  When  operators  choose  small  objects  as  aimpoints,  their 
demand  for  detail  in  DFAD  increases. 

2.  Overall,  the  SAR-experienced  subjects  found  Level  2  unacceptable;  rated 
Levels  Y  (15m)  and  Z  (20m)  as  acceptable  equally  often;  but  preferred  Level  X, 
where  a  small  object  could  be  used  as  an  aimpoint.  At  a  minimum,  therefore, 
DFAD  data  must  contain  a  feature  density  somewhere  between  Level  Z  (20m)  and 
Level  Y  (15m)  to  adequately  simulate  SAR  for  a  majority  of  experienced  users. 

III.  SAR  RADAR  SCOPE  INTERPRETATION 
CUE  ANALYSIS 

Introduction 


Radar  Scope  Interpretation  (RSI)  cues  or  pointers  are  surface  features  that 
serve  as  reflectors  to  aid  radar  operators  in  locating  an  aimpoint.  These 
pointers  are  identified  from  charts  and  photographs  of  the  target  area  during 
pre-mission  study.  Typically,  the  operator  will  select  distinctive  features  to 
provide  overall  orientation  and  then  locate  successively  smaller  features  to 
help  locate  the  target.  For  example,  in  Figure  4,  Greybull,  Wyoming,  the 
circled  building  at  the  center  of  the  photo  is  the  target.  The  operator  would 
first  look  for  the  river  on  the  east  edge  of  town  and  the  railroad  tracks  on 
the  west.  The  operator  would  then  look  for  the  bridge  and  main  road  running 
east  to  west.  The  target  is  therefore  the  first  major  reflector  north  of  the 
main  road  where  it  intersects  the  railroad  tracks. 
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The  results  of  the  Density  Requirements  study  indicated  that  not  all 
features  within  a  scene  are  of  equal  importance  for  SAR  operators.  The  major 
difference  between  Level  X  (10m)  and  Level  Y  (15m)  is  that  features  the  size  of 
houses  within  residential  areas  are  individualy  portrayed  in  Level  X  and 
aggregrated  into  a  single  areal  feature  in  Level  Y.  This  aggregation  is 
illustrated  by  the  housing  area  at  the  north  edge  of  Scott  AFB  in  Figures  7  and 
8.  Over  70%  of  the  SAR  operators  in  the  Density  Requirements  Study  found  Level 
Y  acceptable,  indicating  that  they  did  not  require  depiction  of  structures  such 
as  houses  but  that  the  roads  within  the  housing  area  were  necessary. 

This  RSI  Cue  Analysis  study  was  conducted  to  more  systematically  determine 
what  surface  features  SAR  operators  use  as  pointers.  The  objective  of  the 
study  was  to  identify  the  characteristics  and  locations  of  pointers  relative  to 
the  aimpoint  for  different  types  of  scenes.  These  data  could  then  be  used  to 
specify  what  features  require  ground  truth  depiction  within  a  simulated  SAR 
image  and  what  features  could  be  depicted  generically  without  affecting 
operator  performance. 


Research  Methods 


Overview 


For  their  pre-mission  study,  B-l  OSOs  examine  Fixpoint  Graphic  cards 
similar  to  Figure  4.  The  card  contains  the  geodetic  coordinates  and  elevation 
of  the  specified  fixpoint  and  a  1.25nm  x  1.25nm  vertical  photograph  of  the  area 
surrounding  the  fixpoint.  The  0S0  identifies  RSI  cues  in  the  photograph  so 
that  he  can  quickly  locate  the  target  on  the  radar  scope.  For  the  RSI  Cue 
Analysis,  OSOs  were  asked  to  examine  a  number  of  Fixpoint  Graphics  and  to 
indicate  the  RSI  cues  they  would  use.  These  data  were  then  analyzed  to 
determine  the  type,  size,  and  location  of  SAR  pointers  for  several  types  of 
scenes. 

Materials 


Unclassified  B-1B  Fixpoint  Graphics  were  obtained  for  115  targets  in  the 
Strategic  Training  Range  Complex.  The  115  scenes  were  assigned  to  one  of  four 
categories,  depending  on  the  feature  specified  as  the  fixpoint  and  the 
surrounding  features.  The  categories  were:  Urban,  Small  Group,  Isolated,  and 
Terrain.  The  Fixpoint  Graphics  were  then  divided  into  five  sets  of  23  with 
approximately  equal  numbers  of  Urban,  Small  Group,  Isolated,  and  Terrain  scenes 
in  each  group. 

Subjects 


Twenty-two  SAR-experi enced  Bl-B  OSOs  from  the  96th  BMW,  Dyess  AFB,  Texas 
served  as  subject-matter  experts.  Mean  flight  time  in  the  B-l  for  these  OSOs 
was  110  hours. 

Procedure 


OSOs  were  contacted  at  their  squadrons.  Each  0S0  performed  an  RSI  Cue 
analysis  on  one  of  the  five  sets  of  fixpoint  graphics. .The  analysis  consisted 
of  first  examining  the  Fixpoint  Graphic  and  then  marking  a  photocopy  of  the 
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graphic  with  a  highlighter  to  indicate  the  SAR-relevant  pointers.  The  task  was 
self-paced  and  took  between  10  and  20  minutes. 

Results  and  Pi scussi on 

Each  highlighted  feature  was  assigned  to  one  of  four  pointer  classifica¬ 
tions:  Lines  of  Communication  (LOCs ) --roads ,  railroad  tracks,  power  lines; 
Natural  Features/Vegetation — rivers,  lakes,  treelines,  plowed  fields; 
Structures--man-made  objects  occupying  three-dimensional  space;  and 
Terrai n--elevati on  and  shadow  effects.  The  percentages  of  pointer  types 
selected  for  each  scene  type  are  listed  in  Table  4. 


Table  4.  Classification  of  Pointer  Types  and  Scene  Types 


Scene  Type 


Pointer 

Type 

Urban 

Small 

Group 

I  sol ated 

Terrain 

Total 

LOCs 

69% 

60% 

57% 

10% 

56% 

Natural 

Features 

13% 

19% 

27% 

8% 

20% 

Structures 

17% 

20% 

12% 

1% 

15% 

Terrain 

1% 

1% 

4% 

81% 

9% 

Examination  of  Table  4  shows  that  the  majority  of  features  used  as  RSI  cues 
for  scene  types  other  than  terrain  were  Lines  of  Communication.  Terrain  was 
used  as  a  pointer  only  where  the  aimpoint  itself  was  a  terrain  feature.  In 
scenes  where  there  were  no  visible  LOCs,  99%  of  the  highlighted  RSI  cues  were 
Terrain  features.  For  scenes  in  which  roads  or  railroad  tracks  were  present, 
60%  of  pointers  highlighted  were  LOCs  (i.e.,  lineal  features  that  typically 
extend  across  much  of  the  scene).  Road  intersections  and  grade  crossings  were 
not  scored  as  separate  features;  however,  the  comments  of  the  OSOs  as  they  were 
highlighting  pointers  indicated  that  a  typical  approach  would  be  to  find  the 
major  LOCs  and  then  follow  them  to  an  intersection.  The  aimpoint  would  then  be 
located  relative  to  this  intersection.  RSI  cues  for  Greybull,  Wyoming  (Figure 
4)  cited  earlier  would  be  a  typical  example. 

After  LOCs,  Structures  were  cited  most  frequently  for  Urban  and  Small  Group 
scenes  whereas  Natural  Features/Vegetation  was  the  next  most  common  pointer  for 
Isolated  scenes.  The  types  of  structures  highlighted  were  most  often  the 
largest  or  tallest  structure  within  an  area,  such  as  a  grain  elevator  or  a 
tower.  Since  Isolated  scenes  were  defined  by  the  absence  of  other  man-made 
features,  it  is  not  unexpected  that  Natural  Features  are  commonly  cited 
pointers.  Like  the  LOCs,  these  Natural  Features  tended  to  be  large  and  extend 
across  a  significant  portion  of  the  scene. 
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One  option  in  developing  a  high-resolution  data  base  would  be  to  include 
only  a  small  high-resolution  patch  around  the  aimpoint.  Analyses  were 
conducted  to  determine  if  the  highlighted  types  of  pointer  (e.g.,  Structures 
vs.  LOCs)  varied  as  a  function  of  distance  from  the  aimpoint.  For  features 
such  as  roads  or  rivers,  distance  was  computed  from  the  aimpoint  to  a  location 
halfway  between  the  nearest  and  most  distant  points  on  the  feature.  For  the 
photographs  on  the  Fixpoint  Graphics  which  covered  a  1.25nm  x  1.25nm  patch,  the 
radial  distance  from  the  aimpoint  to  the  closest  edge  is  0.625nm  or  3,800'. 
Distances  from  pointers  to  the  aimpoints  were  classified  into  four  zones:  Onm 
(i.e.,  the  pointer  is  the  aimpoint), less  than  0.125nm  ( 760 ' ) ,  less  than  0.25nm 
(1,520'),  and  less  than  0.5nm  (3,040').  These  data  are  summarized  in  Table  5. 


Table  5.  Classification  of  pointer  type  and  distance  from  the  aimpoint. 
Distance  zones  are  cumulative  so  that  100%  of  highlighted  RSI 
cues  are  within  0.625nm  of  the  aimpoint. 

Distance  Zone 


Poi nter 
Type 

Onm  (pointer 
is  aimpoint) 

<0. 125nm 
(760') 

<  0.25nm 
(1,520') 

<  0.5nm 
(3,040') 

LOCs 

10% 

65% 

86% 

99% 

Natural 

Features 

14% 

53% 

84% 

99% 

Structures 

65% 

92% 

97% 

100% 

Terrain 

18% 

53% 

75% 

98% 

Total 

20% 

65% 

86% 

99% 

The  data  in  Table  5  show  that  the  types  of  SAR  features  used  as  pointers 
change  as  a  function  of  distance  from  the  aimpoint:  92%  of  Structures  which 
were  highlighted  as  significant  RSI  cues  were  less  than  760'  from  the 
aimpoint.  In  the  majority  of  these  cases,  the  Structure  was  the  aimpoint. 

Many  OSOs  stated  that  if  the  aimpoint  was  an  isolated  Structure  (such  as  a 
radio  tower)  or  was  the  tallest  Structure  in  an  area  (such  as  a  grain 
elevator),  the  Structure  would  be  the  brightest  reflector  in  the  scene  and 
serve  as  an  RSI  cue.  Individual  Structures  were  therefore  rarely  used  as 
pointers  unless  they  were  in  the  immediate  vicinity  of  the  aimpoint. 

LOCs  and  Natural  Features,  however,  were  used  as  pointers  throughout  the 
scene.  Further  analysis  of  the  RSI  data  shows  that  the  area  containing  75%  of 
the  highlighted  LOCs  extended  from  the  aimpoint  to  1,155'  while  75%  of  the 
Natural  Features  were  within  1,340';  75%  of  the  highlighted  Terrain  features 
extended  to  1 , 520' . 
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This  analysis  of  RSI  cues  suggests  that  the  amount  of  ground  truth  feature 
density  provided  by  Level  X  and  Level  Y  may  not  be  required  for  the  entire 
scene.  The  smallest  features  highlighted  as  pointers  were  Structures,  and  the 
great  majority  of  these  were  within  the  immediate  vicinity  of  the  aimpoint.  In 
addition,  the  highlighted  Structures  were  most  often  relatively  large  buildings 
such  as  grain  elevators.  Individual  structures  outside  this  area  might  be 
depicted  generically  by  breaking  up  areal  features  such  as  the  housing  area  at 
Scott  AFB  (Figure  5).  Conversely,  LOCs  and  Natural  Features  such  as  roads, 
railroad  tracks,  rivers,  bodies  of  water,  and  areas  of  uniform  vegetation  need 
to  be  portrayed  accurately  throughout  the  scene.  A  simulated  SAR  image 
containing  both  ground  truth  and  generic  features  would  be  less  accurate  than 
Level  X  but  the  loss  in  accuracy  might  not  affect  operator  performance  because 
the  generic  features  would  not  be  task-critical .  The  simulation  would  also 
look  sufficiently  like  SAR  to  ensure  operator  acceptance. 

IV.  HIGH-RESOLUTION  OPTIMIZATION  STUDY 
Introduction 


The  major  conclusion  drawn  from  the  Density  Requirements  study  was  that 
data  base  density  greater  than  current  DMA  DFAD  products  was  required  to 
support  SAR  simulation.  The  RSI  Cue  Analysis,  however,  suggested  that  the 
features  contained  in  Level  2  (e.g.,  large  Structures  and  LOCs)  were  of  major 
importance  in  executing  SAR  tasks.  Features  unique  to  Level  Y  or  Level  X,  such 
as  smaller  Structures,  were  infrequently  cited  as  pointers.  On  the  basis  of 
these  findings,  it  was  predicted  that  Level  2  data  might  be  enhanced  in  ways 
that  would  maintain  SAR-like  appearance  of  the  simulation  and  still  support 
B-1B  0S0  task  performance.  In  particular,  the  large  block  areal  features  in 
Level  2  which  represent  aggregations  of  smaller  reflectors  could  be  broken  up 
generically  to  resemble  the  appropriate  ground  returns  but  without  maintaining 
strict  ground  truth.  Major  LOCs,  however,  would  have  to  retain  ground  truth 
even  within  enhanced  areas. 

To  test  this  prediction,  a  simulator-based  study  was  designed  using  both 
task  performance  and  subject-matter  expert  ratings  as  the  measure  of  simulation 
utility.  Simulated  SAR  images  were  generated  from  enhanced  versions  of 
existing  DMA  products.  Level  1  and  Level  2  DFAD,  and  from  high-resolution  data 
bases,  Level  Y  and  Level  X.  These  simulations,  plus  recorded  SAR  images,  were 
used  by  OSOs  in  a  B-l  simulator  to  locate  a  specified  aimpoint.  Accuracy  of 
crosshair  placement  and  operator  confidence  in  placement,  and  ratings  of 
acceptability  for  mission  rehearsal  for  each  simulation,  were  compared  to 
performance  and  ratings  for  the  actual  SAR. 

Research  Methods 


Overview 


Simulated  SAR  images  were  generated  from  four  data  bases  for  15  scenes. 
Actual  B-1B  SAR  images  were  also  obtained  for  these  scenes.  The  simulated  and 
actual  SAR  were  recorded  for  presentation  in  the  0S0  station  of  the  B-l 
Engineering  Research  Simulator.  SAR -experienced  OSOs  first  studied  Fixpoint 
Graphic  cards  for  the  15  scenes  and  then  used  the  radar  screen  and  track  handle 
in  the  ERS  to  place  the  crosshairs  on  the  target.  Placement  accuracy  and 
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operator  confidence  for  the  different  data  base  levels  were  compared  to 
performance  using  the  actual  SAR.  The  OSOs  were  then  debriefed  to  determine 
what  radar  features  were  contained  in  each  image  and  the  overall  acceptability 
of  the  image  for  mission  rehearsal. 

Materials 


Data  Bases.  For  each  of  15  fixpoints  in  the  Strategic  Training  Range 
Complex,  four  data  bases  were  obtained  from  DMA  or  created  at  ASD/ENET V.  The 
data  bases  were:  Level  1  (100m)  enhanced.  Level  2  (30m)  enhanced.  Level  Y 
(15m),  and  Level  X  (10m).  In  three  of  the  scenes.  Level  Y  contained  block 
areal  features  which  were  aggregations  of  reflectors  that  were  individually 
portrayed  in  Level  X;  these  areal  features  were  enhanced  in  Level  Y  as  well  as 
in  Levels  1  and  2.  Also,  if  the  selected  aimpoint  was  not  present,  it  was 
inserted  into  all  data  bases. 

The  15  Level  Y  and  11  of  the  15  Level  2  data  bases  were  created  at 
ASD/ENETV.  This  was  accomplished  by  filtering  the  DMA  Level  X  data  to  the 
appropriate  level  of  detail.  Some  features  were  retained  regardless  of  size 
because  of  their  reflectivities.  All  objects  with  certain  Feature 
Identification  Descriptors  (FIDs)  (e.g.,  power  pylons)  were  also  retained.  The 
conventions  used  correspond  to  DMA  procedures  for  capturing  features  in  Level  2 
and  are  summarized  in  Table  6. 


Table  6.  Exceptions  to  the  Data  Base  Capture  Criteria 
Used  by  ASD/ENETV  in  Generating  Level  2  and 
Level  Y  Data  from  Level  X. 


1.  All  features  with  a  Surface  Material  Code  (SMC)  of  1  (metal)  or  2 
(part-metal)  were  retained. 

2.  Level  X  features  coded  with  Feature  Identification  Descriptor  (FID)  917 
(cultivated  field)  were  changed  to  FID  902  (normal  soil)  when  creating  Level  2 
data  bases. 


3.  Level  2  FIDs  250  and  251  (roads)  were  included  regardless  ot  \ 

4.  Level  Y  FIDs  240  -  255  (roads)  were  included  regardless  of  size. 

5.  The  following  FIDs  were  retained  in  Levels  2  and  Y  regardless  of  size: 


230  -  239 
260  -  267 
270  -  277 
801  -  807 
820  -  824 
915 


Road  Interchanges 
Bridges 

Bridge  Superstructures 

Tanks 

Silos 

Islands 


6.  All  FID  927s  (fences)  were  retained  in  Level  Y  regardless  of  length, 
while  only  fences  150m  or  longer  were  retained  in  Level  2. 
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A  synthetic  breakup  program  developed  at  ASD/ENETV  was  applied  to  areal 
features  in  Data  Base  Levels  1,  2,  and  Y.  This  program  allows  insertion  of 
roads  and  buildings  as  specified  by  the  user.  Parameters  within  the  breakup 
program  included:  road  pattern  (i.e.,  length,  width,  and  orientation);  building 
density;  and  building  length,  width,  and  orientation.  Appropriate  parameter 
values  were  inserted  manually  based  on  examination  of  aerial  photographs  of 
each  area.  This  process  maintained  generic  accuracy  within  the  enhanced  area 
but  did  not  provide  strict  ground  truth.  Figures  9,  10,  and  11  are  data  base 
plots  for  Greybull,  Wyoming  at  Level  1  enhanced.  Level  2  enhanced,  and  Level  Y 
(no  enhancements),  respectively.  When  compared  with  Level  1  and  Level  2 
without  enhancements  (Figures  1  and  2),  it  is  apparent  that  the  enhancement 
process  has  primarily  added  secondary  LOCs  (residential  streets)  and  small 
Structures  (houses). 


FIXPOINT  #6  LEVEL  I  ENHANCED 


LAT:  44  :  JO  :  24  .  0  N  LAT:  44  :  JO  :  24  .  0  II 

LOK  100:  4  :  42  .  0  «  LOR  100:  1  :  40  .  0  II 


Figure  9.  Plot  of  Digital  Feature  Analysis  Data  for  Greybull,  Wyoming, 
__  at  Level  i  as  Enhanced  for  the  High-Resolution  Optimization 
Study . 
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LATITUDE 


LAY:  44  :  SO :  S4  .  0  N 
LOtt  100:  4  :  43  . 0  M 


FIXPOINT  #6  LEVEL  Y 


LAY;  44  :  SO  :  34  .  0  M 
LOtt:  100:  1  :  40  .  0  If 


LAY:  44  :  St : 
Utt  Sit:  4  : 


•  If 

•  « 


re  11.  Plot  of  Digital  Feature  Analysis  Data  for  Greybull,  Wyoming 
at  Level  Y  Used  in  the  High-Resolution  Optimization  Study. 


Transformation  and  Image  Generation.  Simulated  SAR  images  were  generated 
on  a  General  Electric  DRLMS  located  at  ASD/ENETV.  This  DRLMS  was  developed  to 
simulate  B-52/KC- 135  real  beam  radars  and  has  been  modified  to  process  high- 
resolution  data.  The  transformation  reflectivity  and  radar  gains  were 
optimized  for  each  scene  to  obtain  maximum  correlation  with  the  actual  SAR  and 
the  most  accurate  portrayal  of  the  different  data  bases.  The  simulated  SAR 
images  were  generated  for  a  0.625nm  x  0.625nm  patch  and  formatted  to  match  the 
actual  SAR. 

SAR  Imagery .  SAR  images  for  the  15  scenes  were  obtained  from  a  test  flight 
of  a  BAC-Hl  aircraft  owned  by  Westinghouse  and  used  for  development  of  the 
B-1B  radar.  Aircraft  altitude  was  approximately  10,000'  and  range  was  7  to 
lOnm,  resulting  in  depress ; jn  angles  of  8  to  14  degrees. 

Simulator  and  Study  Software.  The  B-1B  Engineering  Research  Simulator  was 
developed  by  Scientific  Applications  International  Corporation  (SAIC)  under 
contract  with  the  Armstrong  Aerospace  Medical  Research  Laboratory  and  was 
located  at  the  338th  Combat  Crew  Training  Squadron,  Dyess  AFB,  Texas.  Only  the 
radar  screen  and  track  handle  of  the  0S0  station  were  used  for  the  Optimization 
study.  To  appear  on  the  radar  screen,  simulated  radar  images  are  created 
off-line,  stored  on  video  disk,  and  called  up  as  appropriate.  Since  this 
system  allows  storage  of  any  image  on  video  disk,  the  actual  SAR  images  could 
also  be  presented  in  the  simulator.  Using  a  predetermined  sequence,  the  study 
software  called  up  the  appropriate  images  for  each  subject,  controlled  the 
timing  of  events,  and  recorded  the  subject's  placement  accuracy  and  confidence. 

Experimental  Design 

Stimulus  images  consisted  of  SAR  simulations  generated  from  four  data 
bases,  plus  actual  SAR,  for  15  scenes.  The  scenes  were  blocked  into  three 
groups.  Urban,  Small  Group,  and  Isolated,  based  on  the  number  of  features 
surrounding  the  aimpoint.  Each  subject  saw  only  one  level  of  each  scene.  The 
combination  of  scene  and  image  level  was  balanced  so  that  each  subject  saw 
three  scenes  from  each  of  the  four  levels  of  data  bases  and  three  actual  SAR 
scenes.  Stimulus  presentation  order  was  randomized  for  each  subject. 
Twenty-five  subjects  were  required  for  complete  balance.  Since  there  were  only 
five  observations  per  level  per  scene,  data  were  analyzed  by  scene  group 
(Urban,  Small  Group,  Isolated)  rather  than  by  individual  scene.  The  dependent 
variables  were:  (a)  deviation  of  the  subject's  crosshair  placement  from  an 
Optimal  Aimpoint  previously  determined  by  a  Strategic  Air  Command  (SAC) 
subject-matter  expert,  and  (b)  operator  confidence  rating. 

Subjects 


Subjects  were  25  SAR-experienced  B-1B  OSOs  and  instructors.  Median 
experience  was  220  B-1B  hours.  All  subjects  had  over  1,000  total  flight  hours. 

Procedures 


Briefing.  Each  subject  was  tested  individually.  The  purpose  of  the 
experiment  and  the  type  of  data  to  be  collected  were  described  (see  Appendix 
A).  The  0S0  was  then  given  10  minutes  to  study  the  Fixpoint  Graphic  cards  of 
the  15  scenes  in  the  study,  plus  two  scenes  for  warm-up. 
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Warm-Up  Trials .  All  OSOs  received  two  warm-up  trials  using  the  same 
images;  performance  and  confidence  data  were  collected  but  not  retained.  After 
both  warm-up  trials,  there  was  a  pause  for  the  experimenter  to  answer  any 
questions . 

Data  Col  lection.  After  the  warm-up  trials,  data  were  collected  in  three 
sets  of  five  trials,  with  a  break  between  sets.  The  OSO's  task  was  to  locate 
the  designated  aimpoint  as  accurately  as  possible  on  the  radar  screen  within  60 
seconds.  Before  each  trial,  the  0S0  was  given  30  seconds  to  study  the  Fixpoint 
Graphic.  At  the  end  of  the  study  interval,  the  screen  alphanumerics  were 
illuminated  and  a  tone  sounded,  indicating  that  the  0S0  should  request  a  radar 
map.  When  the  map  was  completely  drawn,  the  crosshairs  appeared  in  the  center 
of  the  screen  and  a  response  time  clock  started.  The  0S0  moved  the  crosshairs 
by  depressing  the  track  handle  trigger  and  pushing  a  thumb  control.  When 
satisfied  with  the  placement,  the  0S0  depressed  a  button  on  the  track  handle 
which  blanked  the  screen  and  the  trial  ended.  If  a  target  was  not  selected 
within  60  seconds,  the  screen  blanked  automatically,  erasing  the  image.  A 
warning  tone  sounded  15  seconds  before  the  screen  blanked.  After  the  target 
was  designated,  the  0S0  stated  his  confidence  in  the  placement  on  a  7-point 
scale  ranging  from  "1 — Complete  Guess"  to  "7--Very  High  Confidence. 11  The 
experimenter  entered  this  rating  at  a  terminal  and  started  the  study  period  for 
the  next  trial . 

Debriefing.  After  performance  data  were  collected  for  all  15  stimuli, 
subjects  were  shown  photographs  of  the  images  they  had  seen,  along  with  the 
appropriate  Fixpoint  Graphic.  The  0S0  then  examined  each  photograph  and 
completed  a  questionnai re  rating  the  visibility  of  different  types  of  features 
and  the  acceptability  of  the  image  for  "Procedures  Training"  and  for  "Mission 
Rehearsal"  (see  Appendix  B). 


Results  and  Discussion 


Placement  Accuracy 

Crosshair  placement  accuracy  was  defined  as  the  radial  miss  distance 
between  the  subject's  placement  of  the  crosshairs  and  the  ideal  crosshair 
placement  within  each  image.  The  ideal  crosshair  position  was  defined  using 
two  separate  techniques.  The  first  technique,  previously  described  under 
Experimental  Design,  used  the  Optimal  Aimpoint  designated  by  the  SAC  subject- 
matter  expert.  The  second  technique  defined  the  ideal  crosshair  position  for  a 
given  scene  as  the  mean  of  the  crosshair  placements  designated  by  subjects  when 
viewing  the  actual  SAR  image  of  that  scene. 

To  make  the  variances  more  uniform  across  conditions  and  to  reduce  the 
relative  importance  of  a  few  very  poor  crosshair  placements,  a  logarithmic 
transformation  was  applied  to 'the  miss  distances.  This  transformation  can  be 
expressed  as: 

X'  =  log2  (X  +  1) 

V  =  log2  (Y  +  1) 
d  =  (X'2  +  y ’ 2 ) 1 /2 
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where  X  is  the  untransformed  miss  distance  on  the  horizontal  axis,  Y  is  the 
untransformed  miss  distance  on  the  vertical  axis,  X*  and  Y1  are  the  transformed 
miss  distances  on  each  axis,  and  d  is  the  composite  miss  distance  used  in  the 
analyses. 

Figure  12  shows  the  mean  of  the  transformed  differences  between  the 
subjects'  crosshair  placements  and  the  Optimal  Aimpoint.  The  crosshair 
placement  errors  have  been  scaled  such  that  the  mean  miss  distance  for  Isolated 
fixpoint  SAR  images  equals  1.0.  The  mean  placement  accuracy  for  each  of  the 
other  combinations  of  scene  type  and  data  base  was  then  scaled  relative  to  this 
value  so  that  the  ratio  property  of  all  values  was  retained.  Therefore,  it  is 
mathematically  correct  to  describe  the  mean  transformed  error  for  Level  1 
enhanced  Urban  fixpoints  as  being  twice  as  large  as  the  mean  transformed  error 
for  Level  1  enhanced  Isolated  points. 


Data  Baaa  Density  Laval 


Figure  12.  Mean  Log  Miss  Distance  from  the  Optimal  Aimpoint  for  Each 
Group  and  Data  Base  Level  in  the  High-Resolution 
Optimization  Study.  The  vertical  axis  on  this  figure  is 
scaled  in  arbitrary  units.  Means  within  a  given 
rectangle  were  not  significantly  different  from  each  other 
(p  <  .05). 

Overall  crosshair  placement  was  less  accurate  for  Urban  fixpoints,  and 
Level  1  enhanced  images  produced  less  accurate  placements  than  did  other  data 
base  levels  (£  <.05).  The  miss  distances  were  statistically  analyzed  using  a 
partially  balanced  incomplete  block  design  (Kempthorne,  1979)  analysis  of 
variance.  For  the  miss  distances  shown  in  Figure  12,  this  analysis  shows  that 
data  base  density  level  significantly  affected  the  accuracy  of  crosshair 
placement  for  both  Urban  [F_(4,  90)  =  3.49,  -  .011]  and  Small  Group  fixpoints 
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[F_( 4 ,  91)  =  3.10,  p  =  .019].  No  effect  was  found  for  data  base  with  Isolated 
points  (p  >  .05).  ihe  significant  main  effects  were  further  analyzed  using 
Fisher 's  Protected  Least  Significant  Difference  test  (Cohen  and  Cohen,  1983). 
The  results  of  this  post  hoc  analysis  are  summarized  through  the  use  of 
rectangles  in  Figure  12.  For  each  type  of  fixpoint,  the  means  enclosed  within 
a  given  rectangle  were  not  significantly  different  from  other  means  enclosed 
within  the  same  rectangle  (£  >  .05);  means  enclosed  by  different  rectangles 
were  significantly  different  from  each  other  (£  <  .05). 

A  similar  analysis  using  the  miss  distances  computed  from  the  mean  SAR 
crosshair  placements  showed  that  the  data  base  density  level  significantly 
affected  crosshair  placement  only  for  the  Small  Group  fixpoints  [F(3,  67)  = 
7.57,  £  <  .0005].  No  effects  were  found  for  data  base  on  either  Urban  or 
Isolated  points  (£  >  .20).  Post  hoc  analysis  of  Small  Groups  data  showed  that 
placements  were  significantly  less  accurate  for  Level  1  enhanced  than  for  Level 
2  enhanced.  Level  Y,  Level  X,  and  SAR. 

The  following  conclusions  can  be  drawn  from  these  analyses  of  the 
crosshair  placement  data: 

1.  Location  of  aimpoints  within  Urban  areas  is  a  more  difficult  task 
than  location  of  Isolated  aimpoints  or  aimpoints  within  Small  Groups  of 
reflectors. 


2.  Data  base  density  level  did  not  affect  placement  accuracy  for 
Isolated  aimpoints.  The  isolated  aimpoints  included  objects  such  as  the  end  of 
a  bridge,  or  a  radio  tower  along  the  side  of  a  road.  Since  these  objects  were 
the  most  salient  features  within  the  scene,  the  presence  or  absence  of  other 
smaller  features  did  not  affect  performance. 

3.  Data  base  level  significantly  affected  placement  accuracy  for 
Urban  and  Small  Group  fixpoints.  In  these  cases,  the  small  features  present 
in  the  Level  2  enhanced.  Level  Y,  and  Level  X  data  bases  served  as  contextual 
cues  to  aid  in  locating  the  aimpoint  whereas  their  absence  from  Level  1 
enhanced  images  degraded  performance. 

Confidence  Ratings 

The  confidence  ratings  were  analyzed  using  an  incomplete  blocks  model 
analysis  of  variance.  There  was  a  significant  effect  due  to  data  base  level 
for  the  Urban  scenes  [F ( 4,  82)  =  9.18,  £  <  .0005],  the  Small  Group  scenes  [F(4, 
90)  =  8.59,  £  <  .0005J7  and  the  Isolated”  scenes  [F_(4,90)  =  4.09,  £  <  .004). 
Confidence  was  lowest  for  Level  1  enhanced  imagery  for  all  groups;  there  were 
no  significant  differences  in  confidence  among  the  Level  2  enhanced.  Level  Y, 
and  Level  X.  There  was  a  significant  negative  correlation  between  confidence 
scores  and  miss  distance  (£2  =  .234,  £  <  .05).  The  correlation  between 
confidence  and  miss  distance  indicates  that  the  OSOs  were  accurate  predictors 
of  their  own  performance;  i.e.,  lower  confidence  was  associated  with  less 
accurate  placements.  This  validates  the  use  of  operator  confidence  as  a 
measure  of  simulation  utility  for  this  task.  Confidence  was  not  correlated 
with  B-1B  flight  hours  (r^  =  .012)  nor  were  flight  hours  correlated  with  miss 
distance  (r^  =  .006).  The  lack  of  correlation  between  B-1B  flight  hours  and 
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confidence  or  miss  distance  indicates  that  effects  are  due  to  differences  among 
the  experimental  conditions  and  not  to  differences  among  subjects. 

Debriefing  Questionnai res 

The  data  obtained  from  the  Debriefing  Questionnaires  (Appendix  B)  were 
analyzed  to  determine  the  influence  of  data  base  level  on  the  subjects' 
responses.  First,  the  subjects'  responses  were  recoded  as  either  Low  (0,  1,  or 
2)  or  High  (3  or  4).  Then,  multidimensional  contingency  tables  were  formed  by 
tabulating  the  subjects'  responses  for  each  debriefing  question  for  each  scene 
type  and  data  base  density  level.  Finally,  a  log  linear  analysis  was  used  to 
determine  if  these  multidimensional  tables  could  be  reduced  to  simpler  two-  or 
three-way  contingency  tables  without  significant  loss  of  information.  This 
analysis  revealed  that  two-way  tables  were  adequate  to  represent  the  data. 

The  two-way  tables  indicated  that  the  data  base  level  significantly 
affected  subjects'  ratings  of  major  road  visibility,  small  feature  visibility, 
and  acceptability  for  mission  rehearsal.  The  relationships  between  data  base 
and  subjects'  responses  are  shown  in  Tables  7  through  9. 


Table  7.  Number  and  percent  of  questionnaire  responses  for  the  item, 

"Are  major  roads  visible?"  for  each  data  base  density  level. 

"Are  Major  Roads  Visible?" 

NOT  VISIBLE  CLEARLY  VISIBLE  ROW  TOTAL 


Data  Base 
Density  Level 


Column  Totals 


1 

47 

25 

72 

65.3% 

34.7% 

19.7% 

2 

32 

42 

74 

43.2% 

56.8% 

20.3% 

Y 

23 

50 

73 

31.5% 

68.5% 

20.0% 

X 

27 

47 

74 

36.5% 

63.5% 

20.3% 

SAR 

15 

57 

72 

20.8% 

79.2% 

19.7% 

114 

221 

365 

39.5% 

60.5% 

100% 

26 


Table  8.  Number  and  percent  of  questionnaire  responses  for  the  item, 

"Are  small  features  visible?"  for  each  data  base  density  level. 

"Are  Small  Features  Visible?" 


NOT  VISIBLE 

CLEARLY  VISIBLE 

ROW  TOTAL 

Data  Base 

Density  Level 

1 

66 

7 

73 

90.4% 

9.6% 

19.8% 

2 

53 

20 

73 

72.6% 

27.4% 

19.8% 

Y 

43 

31 

74 

58.1% 

41.9% 

20.1% 

X 

40 

34 

74 

54.1% 

45.9% 

20.1% 

SAR 

29 

45 

74 

39.2% 

60.8% 

20.1% 

Column  Totals 

231 

137 

368 

62.8% 

37.2% 

100% 

Table  9,  Number  and  percent  of  questionnaire  responses  for  the  item, 

"Is  this  image  acceptable  for  mission  rehearsal?"  for  each 
data  base  density  level. 

"Is  This  Image  Acceptable  for  Mission  Rehearsal?" 


NOT  ACCEPTABLE 

ACCEPTABLE 

ROW  TOTAL 

Data  Base 

Density  Level 

1 

62 

12 

74 

83.8% 

16.2% 

19.9% 

2 

45 

30 

75 

60.0% 

40.0% 

20.2% 

Y 

40 

34 

74 

54.1% 

45.9% 

19.9% 

X 

45 

29 

74 

60.8% 

39.2% 

19.9% 

SAR 

25 

49 

74 

33.8% 

66.2% 

19.9% 

Column  Totals 

217 

58.5% 

154 

41.5% 

371 

100% 
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Post  hoc  analyses  of  these  tables  showed  the  following  ordering  (jd_  <  .05) 
between  subjects'  responses  for  the  different  levels  of  data  base: 

-  Visibility  of  major  roads 

Level  1  enhanced  <  Level  2  enhanced  =  Level  Y  =  Level  X  <  SAR 

-  Visibility  of  small  features 

Level  1  enhanced  <  Level  2  enhanced  <  Level  Y  =  Level  X  <  SAR 

-  Acceptability  for  mission  rehearsal 

Level  1  enhanced  <  Level  2  enhanced  =  Level  Y  =  Level  X  <  SAR 

Analysis  of  the  Confidence  and  Debriefing  data  support  the  conclusions 
drawn  from  crosshair  placement  accuracies.  OSOs  had  less  confidence  in 
placements  using  SAR  simulations  generated  from  Level  1  enhanced  data  than  for 
Level  2  enhanced.  Level  Y,  or  Level  X.  Likewise,  the  subjects  reported  seeing 
fewer  features  in  Level  1  enhanced  and  rated  it  significantly  less  acceptable 
than  the  other  Levels  for  mission  rehearsal.  There  were  no  significant 
differences  in  acceptability  for  mission  rehearsal  among  Level  2  enhanced. 

Level  Y,  and  Level  X;  however,  it  should  be  noted  that  all  three  levels  were 
rated  as  less  acceptable  than  actual  SAR  images. 

Comparison  with  Density  Requi rements  Study 

In  the  Density  Requirements  study.  Level  2  was  rated  by  OSOs  as  acceptable 
for  simulating  SAR  on  only  10%  of  the  trials  while  Level  Y  was  rated  acceptable 
on  82%  of  trials  (see  Table  3).  In  the  present  study,  no  significant 
differences  were  found  in  task  performance,  confidence  in  placement  accuracy, 
or  rated  acceptability  for  mission  rehearsal  between  Level  2  enhanced.  Level  Y, 
and  Level  X.  There  are,  however,  pronounced  differences  between  the  simulated 
SAR  images  used  in  the  two  studies. 

For  the  Density  Requirements  study,  simulations  were  produced  without 
breakup  and  with  minimal  texturing.  This  meant  that  reflectors  such  as  houses, 
small  buildings,  or  minor  roads  were  not  depicted  in  Level  2  imagery.  The 
smaller  capture  criteria  for  Level  Y  and  Level  X  allowed  these  features  to  be 
imaged  individually  so  that  these  simulations  appeared  more  like  SAR  than  the 
simulations  based  on  Level  2.  In  the  present  study,  Level  1  enhanced  and  Level 
2  enhanced  contained  additional  information  on  roads  and  buildings  obtained 
from  aerial  photographs  and  not  from  DMA  data  base  products.  The  synthetic 
breakup  and  texturing  added  to  the  Level  2  enhanced  images  appear  to  have  been 
sufficient  to  improve  realism  and  allow  the  OSOs  to  perform  the  task.  It 
should  be  noted  that  the  realism  in  Level  Y  and  Level  X  is  obtained  from  ground 
truth  while  the  realism  in  the  Level  2  enhanced  images  is  partially  generic. 

The  information  in  Level  1  DFAD,  however,  is  so  sparse  that  adding  enhancements 
did  not  produce  acceptable  simulations. 

Conclusions 


The  following  conclusions  can  be  drawn  from  the  Crosshair  Placement 
Accuracy,  Confidence,  and  Debriefing  data  for  the  High-Resolution  Optimization 
study: 
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1.  A  satisfactory  simulation  of  SAR  was  not  created  from  Level  1  DFAD. 

The  addition  of  synthetic  breakup  did  not  add  sufficient  information  to  Level  1 
to  support  adequate  task  performance. 

2.  No  significant  differences  in  task  performance,  operator  confidence,  or 
acceptability  for  mission  rehearsal  were  found  among  the  Level  2  enhanced, 

Level  Y,  and  Level  X  images. 

3.  For  the  aimpoints  used  in  this  study,  the  highly  detailed  ground  truth 
information  contained  in  Level  X  did  not  produce  better  task  performance, 
operator  confidence,  or  acceptabi lity  for  mission  rehearsal  than  did  the  less 
detailed  ground  truth  in  the  Level  2  enhanced  data  base. 

4.  Neither  training  nor  transfer  of  training  was  addressed  in  this  study 
since  task  performance  was  assessed  using  experienced  SAR  operators  as 
subjects . 


V.  CONCLUSIONS 

The  data  for  these  three  studies  were  collected  from  highly  experienced  SAR 
operators  and  reflect  expert  evaluation  and  task  performance.  No  data  were 
collected  from  novices  training  to  be  SAR  operators,  nor  was  transfer  of 
training  from  simulation  to  actual  SAR  assessed.  Constraints  of  time, 
personnel,  and  ongoing  training  schedules  prevented  the  use  of  -aining  or 
transfer-of-training  designs.  The  studies  conducted  were  based  on  the 
hypothesis  that  simulations  which  are  acceptable  to  SAR  experts  will  contain 
the  task-critical  elements  of  SAR  imagery  required  to  support  training  of 
novice  SAR  operators.  It  is  possible  that  there  are  training  tasks  for  which 
high-resolution  ground  truth  data  bases  are  required;  however,  no  attempt  was 
made  to  identify  such  tasks. 

The  previously  cited  engineering  analysis  conducted  by  TASC  (1985) 
concluded  that  acceptable  SAR  simulations  would  be  supported  by  Level  X  (10m) 
DFAD.  This  conclusion  was  supported  by  the  ratings  from  SAR  subject-matter 
experts  in  the  Density  Requirements  study.  Further,  these  data  show  that  the 
capture  criterion  could  be  relaxed  from  10m  to  15m  without  affecting  the 
acceptability  of  the  simulations.  However,  the  depiction  of  high  density  areas 
as  single  areal  features  in  Level  2  did  not  produce  acceptable  simulation 
fidelity.  In  contrast,  the  SAR  Radar  Scope  Interpretation  Cue  Analysis  showed 
that  individual  reflectors  within  high  density  areas  are  rarely  used  as  cues 
and  that  ground  truth  information  about  these  features  may  not  be  task- 
critical.  Although  such  an  area  must  have  the  appearance  of  the  appropriate 
ground  returns  and  not  uniform  brightness,  the  reflectors  may  be  depicted 
generically  and  still  maintain  acceptable  simulation  fidelity  without  affecting 
task  performance.  Other  features  such  as  major  roads,  rivers,  and  other  lines 
of  communication  are  task-critical  and  must  be  depicted  accurately.  The 
results  of  the  High-Resolution  Optimization  study  confirmed  this  prediction. 

The  information  about  task-critical  features  contained  in  Level  1  was 
insufficient  to  support  0S0  task  performance  even  though  the  data  were 
enhanced.  Level  2,  however,  contains  sufficient  information  about  task 
critical  features  to  support  0S0  task  performance  when  enhanced  with  generic 
information  about  high  density  areas.  The  increases  in  feature  density 
provided  by  Level  Y  or  Level  X  are  not  task-critical. 
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Based  on  the  data  from  the  present  investigations,  the  f o 7  lowing 
conclusions  may  be  drawn: 

1.  Line  features  such  as  roads  and  rivers  as  depicted  in  DMA  Level  2 
DFAD  are  task-critical  in  simulating  SAR. 

2.  SAR  simulations  must  contain  the  appearance  of  SAR  throughout  the 
scene.  Depiction  of  clusters  of  reflectors  as  an  area  of  uniform  brightness  is 
not  acceptable. 

3.  Ground  truth  information  about  features  smaller  than  those 
depicted  in  Level  2  is  not  task-critical.  Generic  portrayal  of  high  feature 
density  areas  will  provide  acceptable  SAR  simulation. 
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APPENDIX  A--Briefing  for  High-Resolution  Optimization  Study 

1.  The  96th  BMW  is  cooperating  with  Aeronautical  Systems  Division  and  the 
Air  Force  Human  Resources  Laboratory  in  a  study  on  simulation  of  Synthetic 
Aperture  Radar.  The  results  of  this  study  will  help  SAC  to  specify  the  data 
base  requirements  for  SAR  simulation  in  the  B-l  Weapon  System  Trainer.  Your 
help  as  an  expert  on  SAR  imagery  interpretation  is  requested.  In  this  study, 
vou  will  use  the  ERS  to  view  a  number  of  actual  and  simulated  SAR  images.  You 
aie  to  examine  each  image  and  to  then  designate  the  aimpoint  as  quickly  and 
accurately  as  possible. 

2.  For  this  study ,  your  only  task  is  to  study  a  fixpoint  card,  request  a 
single  radar  map,  and  then  to  designate  the  aimpoint.  The  scenes  are 
independent  events  and  not  parts  of  a  mission.  Only  the  radar  scope  and  track 
handle  will  be  active;  all  other  systems  will  be  illuminated  but  not 
functional.  Alphanumeric  data  on  the  radar  scope  are  not  the  actual  SAR  image 
parameters  and  should  be  ignored. 

3.  Before  entering  the  ERS  you  will  be  given  10  minutes  to  study  17 
Fixpoint  Graphic  cards.  Once  inside  the  ERS,  the  first  two  trials  will  be  for 
practice  and  the  remaining  15  will  constitute  the  study.  Before  every  trial, 
you  will  have  30  seconds  to  review  the  fixpoint  card.  Then,  the  screen 
alphanumerics  will  be  illuminated  and  a  tone  will  sound  indicating  that  you 
should  request  a  map.  Place  the  crosshairs  onto  the  specified  aimpoint  as 
accurately  as  possible.  When  you  are  satisfied  with  the  placement,  designate 
by  pressing  the  button  to  request  a  second  map.  It  is  important  that  you 
designate  as  soon  as  you  are  satisfied  with  your  placement  since  this  stops  a 
response  time  clock.  If  you  do  not  designate  an  aimpoint  within  60  seconds, 
the  screen  will  blank  and  the  next  trial  will  start.  After  designating  the 
aimpoint,  please  rate  your  confidence  in  placement  accuracy  using  the  following 
scale: 


1  2  3  4  5  6  7 

I _ I _ I _ j _ I _ I _ I 

Complete  Very  High 

Guess  Confidence 

The  experimenter  will  enter  your  rating  and  start  the  30-second  study 
period  for  the  next  trial.  The  experiment  will  pause  after  the  two  practice 
trials  for  you  to  ask  any  questions.  The  remaining  trials  will  be  presented  in 
groups  of  five. 

4.  After  data  collection,  we  will  ask  you  to  evaluate  the  imagery. 

5.  The  data  you  provide  will  be  of  great  value  to  Aeronautical  Systems 
Division  and  the  Defense  Mapping  Agency  in  developing  the  data  base  which  will 
support  the  B-1B  Weapon  System  Trainer.  Thank  you  for  your  help. 
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APPENDIX  B--Debrief ing  Questionnaire  for  High-Resolution  Optimization  Study 


IMAGE  # 


NAME 


# 


DATE 


Compare  this  image  to  what  you  can  see  in  the  fixpoint  graphic  and  to  what  you 
would  expect  to  see  on  the  B-l  SAR. 

1.  Is  the  fixpoint  visible  on  the  radar  image? 


6 

i 

2  3 

4 

Not  Visible 

Clearly  Visible 

Are 

major  roads  visible? 

i _ 

-  1  - 

_ _  i _ i . 

_ i 

0 

i 

2  3 

4 

No  Roads 

Some  Roads 

Most  Roads 

Visible 

Visible 

Clearly  Visible 

How 

important  are  major 

i _ 

roads  in  locating  this 

i  .i  i . 

aimpoint? 

_ i 

6 

i 

2  3 

4 

Not  Important 

Critical 

Are 

large  features  such 

i _ 

as 

_  i . 

rivers,  airfields,  or  towns  visible? 

!  1  1 

6 

i 

2  3 

4 

No  Large 

Some  Large 

Most  Large 

Features  Visible 

Features  Visible 

Features  Visible 

How 

important  are  large 

1 

features  in  locating  this  aimpoint? 

i  i  i  i 

0 

1 

2  3 

4 

Not  Important 

Critical 

Are 

small  features  such 

i _ 

as 

_  i . 

houses,  tanks,  or  towers  visible? 

!  1  1 

6 

i 

2  3 

4 

No  Small 

Some  Small 

Most  Small 

Features  Visible 

Features  Visible 

Features  Visible 

How 

important  are  small 

i _ 

features  in  locating  this  aimpoint? 

i  i  i  i 

6 

i 

2  3 

4 

Not  Important 

5.  Would  this  image  be  acceptable  as  a  simulator  for: 


Critical 


Procedures  Training 


Mission  Rehearsal 


i _ 

6 

Unacceptable 


Unacceptable 


1 


,  i . 


- !  ■ 
2 


N/A 


N/A 


N/A 


Satisfactory 


.  i _ i 

3  4 

Fully  Acceptable 


1 
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Sati 


is^actory 


i 


Fully  Acceptable 
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